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^ ' Abstract 

Q . We consider some certain nonlinear perturbations of the stochastic hnear-quadratic 

I optimization problems and study the connections between their solutions and the cor- 

' responding Markovian backward stochastic diferential equations (BSDEs). Using the 
methods of stochastic control, nonlinear partial differential equations (PDEs) and BS- 

I DEs, we identify conditions for the solvability of the problem and obtain some regularity 

Q I properties of the solutions. 
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^ 1. Introduction 

■ Stochastic optimal control problems and their connections with nonlinear PDEs and back- 

CS| ! ward stochastic differential equations (BSDEs) have been subject of extensive research thanks 

to their wide range of potential applications in engineering, financial economics and related 
areas. However, explicit solutions to such problems can be obtained only in a few special 
cases where the state and control variables usually appear linearly in the state dynamics, 
for example, the linear-quadratic regulator (LQR) problems. Both theoretical and compu- 
tational issues arise when some nonlinear terms are added to the system, sometimes rep- 
resenting the effect of a sudden outside force (perturbation) to the variable of interest. In 
this paper, we consider nonlinear perturbations only in the drift term of a state variable, 
study the properties of the solution to the corresponding control problem and then compare 
it with the standard (unperturbed) LQR problems. Our approach involves using the connec- 
tions between some quasilinear PDEs and a class of forward-backward stochastic differential 
equations (FBSDEs) of the following Markovian form 

dX{t) = fi{t,X{t))dt + a{t,X{t))dW{t), <t <T 

dY{t) = ~F{t,X{t),Y{t),Z{t))dt + Z{t)dW{t), <t <T (1) 
X(0) = x; Y{T)=g{X{T)) 
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where /x and a are the drijt and diffusion terms, respectively, of the forward process X; 
F is the driver term of the backward process Y, and Y{T) = g{X{T)) is the terminal 
condition. We refer the reader to the books by Yong and Zhou (1999), Ma and Yong (1999) 
and the survey paper by El Karoui et. al. (1997) for the general theory and applications 
of backward stochastic differential equations (BSDEs). The existence-uniqueness results for 
nonlinear BSDEs were provided by Pardoux and Peng (1990), Mao (1995), Lepeltier and 
San Martin (1997, 1998), Kobylanski (2000), Briand and Hu (2006, 2008), and Cetin (2012), 
among others. 

The connections between decoupled FBSDEs and quasilinear PDE's were first stated by 
Pardoux and Peng (1992), and Peng (1992) by generalising Feynman-Kac representation of 
PDE's. A version of their results that is relevant to the system ([1]) is given below: 

Theorem 1 (Pardoux and Peng, 1992) Consider the following parabolic PDE: 

vt + ^-Vx + F{t, X, V, avx) + ^cr^x'x' = (2) 

v{T,x) = g{x), 

together with the decoupled system of FBSDEs If the PDE ^ has a (classical) solution 
V, then the pair {¥, Z) with r^'^(t) = t;(t, X^'^(t)), and Z^'^^it) = (T(t, X^'^(t))t;^(t, X^'^(t)) 
solve the BSDE in (C]). Conversely, if the system (Qp has a unique (adapted) solution, then 
v{t,x) = y*'^'(t) is a viscosity solution to the PDE Moreover, this solution is unique if 
the coefficients involved are uniformly Lipshitz. 

We provide the basic definitions and the notations of the paper below. 
1.1 Definitions and Notations 

We consider the one- dimensional Euclidean space M, fixed time- horizon [0,T] and a proba- 
bility space (f2, F, P) where F = {Ft : < t < T} is the complete cr— algebra generated by 
a Brownian motion process W . We define the following function spaces: 

• C^'''([0,T] X R): The space of all real-valued measurable functions / : [0,T] x M such 
that f{t,x) is p (respectively, q) times continuosly differentiable with respect to t 
(respectively, x) where p, q are non-negative integers. 

• L^y(fi): The space of F T-measurable random variables H such that < oo. 

• L'^^{Q): The space of F T-measurable essentially bounded random variables. 

T 

• L^fr{[0,T]): The space of F -adapted processes / such that E[f \ f(t)\''dt] < oo. 



• L^([0,T]): The space of F -adapted essentially bounded processes. 

• Sf^ {C[0, T]): The space of F -adapted continuous processes such that E[ sup |/(t) |^ dt] < 

0<t<T 

oo. 
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For a deterministic function h{t, x) : [0, T] xM — )■ M, the subscript notation denotes partial 
derivatives: ht{t,x) = ^{t,x), hx{t,x) = and hxx{t^x) = x). In particular, for 

time dependent functions or ODE's, dot (') designates the derivative with respect to time 
parameter t. The notation Et[.] will denote the conditional expectation When the 

initial value of a process X is given at time t, then refers to E[.] with Xt = x. We also 

use the following facts and notation related to the stochastic control theory. For the details 
and the proofs of these arguments, one can refer to the books by Fleming and Rishel (1975), 
Fleming and Soner (2006), or Yong and Zhou (1999). 

Let [/ C R and xq G M. Then for [/-valued, F t-adapted control processes Ut, consider the 
following control dependent SDK: 

dXt = a{t,Xt,ut)dt + a{t,Xt)dWt (3) 

Xq = Xq 

Consider also the cost functional 

T 

r{s,x) = Es,x[J f{t,Xt,Ut)dt + g{XT)] (4) 

s 

where the running cost F, the terminal cost g and the control u belong to some appropriate 
Lp or space. We then define the value function as 

V(s,x) = inf J"(s,x) (5) 
where W*'^ is the set of all admissible controls, which consists of all {F t,0 < s < t < T} 

T 

adapted processes u = {u{t), s <t <T} with values in M such that E[f \u(t)\ dt] < oo and 



the state process X*'^'" = X in has a unique strong solution in Lp. When there is no 
ambiguity, the notations X"*'^'", W([0,T],M) will be abbreviated as X and U, respectively. 
We use subscript notation Xt and Ut for the state and control processes when the context is 
clear. If a pair (X*, m*) is optimal for the problem ([3])-([5]) and the value function satisfies V G 
Ci'2([0,T] X M), then by dynamic programming principle (DPP) and standard verification 
theorems, V solves the following second-order nonlinear PDE: 

= mf{f{t,x,u) + {l^v){t,x)} (6) 

u 

with terminal condition v{T, x) = g{x), where L is the backward evolution operator 

L"t;(s, x) = Vs{s, x) + a{s, x, u)vx{s, x) + ^a'^v^xis, x). (7) 

The equation is called the Hamilton- Jacobi-Bellman (HJB) equation (or HJB PDE). The 
PDE is called uniformly parabolic if 3 c > such that |(T()f:, x)\ > c for all {t, x) G [0, T] x R. 
Such PDEs are known to have unique classical solutions under some regularity and growth 
conditions, see for example, Fleming and Soner (2006, IV.4) or Yong and Zhou (1999). 
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Notation 2 We write u G oru&Uto refer to an admissible control system {Q, F, P, X{.),u{.)) 
when the context is clear. 

The next section briefly describes how such a system of FBSDEs can be used to study 
the properties of the solutions to some certain quasihnear HJB PDEs corresponding to the 
stochastic optimal control problems of the form ©-(ITj) where only the drift term of the 
state process is control-dependent. Such an example is the linear-quadratic regular (LQR) 
problems where the diffusion term is control-free, and the value function has an explicit 
(quadratic) form which can be solved analytically or numerically. However, when the drif 
term of the state equation is not linear, an explicit form of the value function may not be 
available since the corresponding HJB equation doesn't have an analytic solution, in general. 
This occurs in the perturbed LQR problems with the drift coefficient having extra nonlinear 
terms. A class of such nonlinear perturbation were studied by Tsai (1978) without a terminal 
cost term. See also Nishikawa et al. (1976). For more general cases, some generalized (e.g. 
viscosity) solutions should be considered. Even if a smooth solution exists, there are some 
other issues to consider: Uniqueness, regularity properties and the numerical computation 
of the solutions. 



2. The PDE and FBSDE Representations 

In this section, we first assume that the stochastic control problem (I3])-(|5]) is solvable with the 
value function V{t,x) G C^'^([0,T] x M), and state a general (weak) representation formula 
for the corresponding FBSDE system, in the spirit of Theorem [TJ However, we will consider 
strong solutions in the rest of the paper. 

Lemma 3 Let V{t,x) G C'^''^{[0,T] x M) be a solution to the optimization problem Ij^- 
1^ and u* = argmin{/(t, X, m) + a{t,x,u)Vx{t,x)} which depends on t and x determinis- 

u 

tically through u*{t,x) = 7^{t,x,Vx{t,x)), representing an optimal Markovian control rule 

u*{t,Xt). Moreover, assume that for some p > 2 and for all s G [0,T), the stochastic in- 

t 

tegral equation X^'^ = x + J cr{r, Xr)dWr has a weak solution (X,W,F ) in L?([s,T]) and 

s 

f{t,X,7i{t,X,V,{t,X)eLl{[s,T]). Then 
(i)V solves the PDE 

vt + ^a^v^r,{t,x) + F{t,x,av^{t,x)) = 

v{T,x) = g{x), 

where F{t, x, z) = a{t, x, n{t, x, cr'^lt, x)z) + f{t, x, 7r(t, x, (T~^{t, x)z). 
(a) A solution to the system 

T T 

rr = giXr) + j F{r, Xr, Zr)dr - j Z^dWr (8) 
t t 

is given by Y'''' = V{t,X''''), Zt'"" = aV^it^X^'"'). 
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Proof. The part (i) follows from the stochastic optimal control and DPP arguments in 
the previous section. For part (ii), let the operator be as in ([7]) corresponding to the 
SDE jH]). Then applying Ito's rule to Y{t) = V{t,X{t)) and using (E]) with u* = u*{t,X) = 
7r{t, X, aVx{t, X), we get: 

dY = {I'^'Vit^X) -a{t,X,n{t,X,Vx{t,X))}dt + a{t,X)Vx{t,X)dW 
= -{fit, X, 7r(t, X, V^it, X)) + ait, X, nit, X, V^it, X))}dt + ZdW 
= -Fit,X,Z)dt + ZdW, 

which also corresponds to the BSDE representation in ([1]) and ([2]), with = ■ 

Under some standard regularity and growth conditions (Lipshitz parameters, linear growth 
etc.), the HJB PDEs (and the corresponding BSDEs) have unique solutions. However, the 
equations that we consider are quite general and there is no guarantee that a (classical) 
solution exists or if it exists whether the solution is unique (in a suitable space). In the next 
subsection, our particular interest will be on a more specific modeling of the state variable X 
and the cost functional /, which has some interesting applications in economics (controlling 
macroeconomic variables) and in engineering (target tracking). We keep the model coeffi- 
cients as general as possible in this section to motivate the choice of the particular functions 
used in the next section. 



2.1 The Additively Separable Drift and Cost Functions 

We consider the following controlled state process X = X", for < t < T : 

dXt = ifxit,Xt) + Bit,Xt)ut)dt + ait,Xt)dWit), 
Xo = xo. (9) 

where the deterministic functions Bit, x), /i(t, x) and ait, x) are continuous functions of their 
arguments. Moreover, for (t, x, m) G [0, T) x M x M given, we introduce the following running 
cost function in (jl]): 

fit,x,u) = hit,x) + kit,x)u^ (10) 

where kit,x) is a positive continuous function on [0, T] x M, and the terminal cost gix) is 
such that giXr) G Lp^. 

Again, by a heuristic application of DPP as in ([6]), the value function Vis, x) in equation 
([5]) satisfies the HJB equation 

= inf{/(t,x,u) + (L"F)(t,x)} 

u 

= Vt + ^(T^V,,it,x) + hit,x)+fiV,it,x) (11) 
+ mi{kit, x)u^ + Bit, x)uV^}. 

u 

^Note that the Brownian motion process W in equation (|8]) is not necessarily the original one due to the 
weak representation approach. 
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Clearly, the minimum of k{t, x)v? + B{t, x)uVx in flTT]) is -^i^^iYxY with the minimizer 

u* = 7rit,x,Vx{t,x)) = ^^^Vx. (12) 

Plugging f[T2l) into the equation ffTTl) . the HJB PDE takes the form of 

1 B'^(t x) 

vt + -a'^Vxx{t,x) + h{t,x) + fiVx{t,x) - j^^j^{vxf = (13) 

v{T,x) = g{x) 

which can also be written as 

vt + ^(r'^Vxx(t,x) + ^Vx(t,x) + F(t,x,avx{t,x)) = (14) 

viT,x) = g{x) 

with 

F{t, X, z) = h{t, x) - Ei^z^^ for (t, x, z) G [0, T] x M x M, (15) 

B^(t,x) 



where H(t, x) 



2k{t,x)c7'^(t,x) ■ 



Remark 4 (a) In a standard LQR problem, the state dynamics in ^ has the usual linear 
drift and diffusion coefficients, k{t,x) = k{t) and B{t,x) = B{t). Moreover, h{t,x) and g{x) 
are quadratic cost functions. It is well known that the condition k{.) > (k{.) > 0, respec- 
tively) is a necessary (sufficient, respectively) condition for the standard LQR problem to be 
solvable. 

(b) If the value function solves the quasilinear PDE [T^) . then the PDE representation given 
by [l4\)- fTS\) can be utilized for a BSDE representation of the problem by Theorem Q] or 
Lemma{3 



2.2 The FBSDE Interpretation 

In view of Theorem [T]( and using the similar steps as in Lemma[3]), if the value function V{t, x) 
is a smooth solution to the equations ffni) - ffT51) . then the pair (1^'^'^, Z^'^) with Yt = V{t,Xt) 
and Zt = a(t, Xt)Vx(t, Xt) is a solution to the BSDE 

dY^s,x ^ -F{t,Xt,Zt)dt + ZtdWt (16) 

with 

t t 
X't'" = ^ + j /^(^' ^r")dr + j a(r, X;'^)ciiy„ (17) 

s s 

and F{t,x,z) as in f[T^ . However, in general, we don't know if the PDE has a classical 
solution since the function F{s, x, z) doesn't satisfy the usual Lipshitz or linear growth 
conditions in the state variables x and z. It also gets more complicated when the terminal 
condition gi^Xx) is not bounded. 



6 



Remark 5 (a) The representation of l[TS\) -ljiTl\) as a FBSDE system is not unique. Another 
representation (as in Lemmal^ may be given by the following system, by eliminating the 
drift term of the forward process: 



t 



s 

T 

F/'^ = g{XT) + j F{r,Xr,Zr)dr - j ZrdWr 



T T 



t t 



where the new driver function is F{t^x,z) = F{t,x,z) + f|ffy2;. Each representation has 
some advantages depending on the complexity level of the forward and backward equations 
in in^) - l[T^) . In this section, the representation [T^) will be used frequently based on the 



assumption that the forward state dynamics {11} has a unique solution. 



(b) When the diffusion term a is time dependent only, the uniqueness of the solutions to 
fT^) can be shown even for more general drift terms, as in Cetin (2012a). For the systems 
with state- dependent diffusion terms a(t,x) and nonlinear drift terms, some more regularity 
or monotonicity assumptions would be needed. 



When the expressions cr(t, x) and x) are time- dependent only, we have the following 
Theorem which is a special case of a result from Cetin (2012) where the driver F is also 
allowed to depend on y: 

Theorem 6 Assume that 

(i) the SDE ( (7?| ) has a unique solution X in L|.[0,T] with a.s. continuous paths 
(a) the value function V{t,x) in ^ is a smooth solution of (T^- (TB) 
(Hi) a(t,x) = o"(t), satisfying |cr(.)| > 6 > uniformly on [0,T]. 

(iv) the function H(t,x) = H{t) is differentiable, and is such that 

[o,r]. 

Then the BSDE [W) has a unique solution ("K, Z) in L|,^ x L\. 

Remark 7 Since the term h{s,X{s)) is not bounded, the existence of a solution is not 
guaranteed in general. However, if a function v{t,x) satisfies the HJB PDF [T^-pR). then 
thanks to the monotonic transformations Ut = exp{—H{t).Y) and At = —H{t)UtZt, the pair 
[exp{-H{t).v{t, Xt), -H{t)a{t)v^{t, Xt) ex-p{-H{t).v{t, Xt))] solves the BSDE with < Ut = 
exp{-H{t).v{t,Xt) < 1. 

Corollary 8 // the value function V(t,x) satisfies the HJB PDF [T^-(TR). then it is the 
unique solution of ^^-^^(j. 



H{.)/H{.) 



IS 



bounded on 



2.3 A Relevant LQR Problem with State-Independent Diffusion 

Now consider the linear state dynamics /i(t,x) = A{t)x and cr(t,x) = a{t) in addition to the 
quadratic cost functions f(t,x,u) = e~^^[{x — ^(t))^ + ki{t)u^] and g{x) = k2{x — ^(T))^, 
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where all the time-dependent functions are continuous. When ^(.) = 0, the problem reduces 
to the standard LQR problem. By an appeal to the standard verification theorems for the 
stochastic control problems (Fleming and Soner, 2006), the system ( !T4l) - (fT5ll can be shown 
to have a unique smooth solution v{t,x), to the corresponding LQR optimization problem 
in the following form: 

V{t,x) = P{t)x^ + K{t)x + N{t) (19) 
V{T,x) = Ux-aT))\ 

where, for s < t < T, the functions P, K and solves the system of the ODEs below: 

0, P{T) = A;2 > 0, (20) 

0, K{T) = -2e-"^e(T), (21) 

0, A^(r) = e-^^C^(T). (22) 

using the notation f{t) = for time derivatives. It is well known (e.g., Fleming and Rishel, 
p. 89) that the Riccati ODE fl20l) has a non-negative (positive if k2 > 0) solution P(.) G 
C^[0,T]. Consequently, the linear first-order equations fl2T|) and fl22|) also have unique solu- 
tions in C^[0,T]. Moreover, an optimal control process is given by 

and the optimized state process which is given by the SDE 

dX; = {A{t)X; + B{t)u{t,X;))dt + a{t)dWt, (23) 

= mt) - ^^^(i) - e'^^imndt + a{t)dw, 

is a Gaussian process on [0, T]. 

The case with a{t,x) = a{t)x is also similar: The solution is a quadratic function of x 
with time dependent parameters being solutions to ODEs similar to those above but we are 
not going to provide the details here (note that the Theorem [6] doesn't apply directly in 
this case). In both cases, the value function can be obtained by solving the corresponding 
ODEs numerically. This can be done efficiently even in high dimensions so it is not necessary 
to consider a FBSDE approach to solve such problems^ However when there is no explicit 
solution available, then the FBSDE approach could be preferable, especially when the cor- 
responding control problem or when the PDE involves a state variable in higher dimensions. 
We describe such a nonlinear application which is a generalized version of an example from 
Tsai (1978). 

^Actually, it is shown in Cetin (2006) that solving a corresponding FBSDE system will result in a sequence 
of iterations which are equivalent to solving the related ODEs above using Euler discretization. 



P(t) + e-^* + 2AP(t) - e^'^P\t) = 

h 

kit) + {A{t) - e^'^P{t))K{t) - 2e-''m = 

N{t) + a^P{t) - ^^K^{t) + e-^'eit) = 
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3. Nonlinear State Dynamics with State-Independent 
Diffusion 



Now consider a non-linear drift term while diffusion coefficient is still depending only on 
time: 

dXt = [A{t)Xt + 6r{t,Xt) + B{t)ut]dt + a{t)dW{t), (24) 
Xo = xo > 

where a{.) is bounded away from on the interval [0, T], p{t, x) is a non-linear perburtation 
term, and 6 is the perturbation constant. The system reduces to a linear one when 6 = 0. 
Let the cost functional for this perturbed problem be given 

T 

r{s,x) = Es,.J [l{t){Xt-at))'' + Htyt)]dt (25) 

s 

where k{.) > and ^(.) is continuous on [0,r]. Define the value function as V{s,x) = 
inf^jgiR J"(s, x), and let the control set U consist of all square integrable adapted processes 
Ut such that the equation (12^ has a unique solution in L^([s,T],R) (however it is sufficient 
to consider only the feedback controls of the Markovian form). This quadratic optimization 
problem cannot be solved explicitly unless 6 = however assuming that the SDE fl24|) has a 
solution for a sufficiently rich set of the control processes, including the candidate optimal 
control u* = -^^Vx (from f|T2l) ) and u = 0, and the optimization problem (jl]) is solvable, we 
can identify a corresponding FBSDE system to characterize the solution. Since the terminal 
condition is bounded (in this case zero, for simplicity), using the methods of the parabolic 
PDEs and stochastic analysis, it can be shown to have a smooth solution (as in Tsai, 1978). 
For more general state equations or terminal conditions, one can only expect to get a less 
smooth (viscosity) solution, using either methods of the PDEs (as in Fleming and Soner, 
2006) or those of the FBSDEs. We know state some assumptions that will be used for the 
main results of this section: 

Condition 9 Consider the equations (2^-^2^ and let fi{t, x) = A{t)x+6r{t, x), Ji{t, x, u) = 
/i(t,a;) + B{t)u. 

(i) The time dependent functions A,B,k,l and a are continuous on [0,T]. 

(a) The function l{t) is nonnegative, k{t) is positive on [0, T] and is bounded away from zero: 

For some e > 0, k{.) > e. 

(Hi) The function H(t) = ^^^s^Jaj^) is continuously differentiable in (0, T) and is bounded away 
from zero. 

(iv) (Monotonicity condition) For all t G [0,T] and x G M, xr{t,x) < C(l + x^), for some 
constant C > 0. 

(v) (Monotonicity condition): For all t G [0,T] and x,?/ G M, (x — y)r(t,x) — r{t,y) < 
K{x — y)"^, for some constant K > 0. 
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Lemma 10 Assume that the functions A,B and a satisfy the Condition\^ (i), r{t,x) is 

locally Lipshitz and satisfies the Condition\E (iv) above. Then, 

(a) For every initial condition Xs = x, the control-free state equation 



X^'- = x+ [A{t)X-:^^ + 5r{t, X^''')dv + / aiv)dW, 



(26) 



has a unique strong solution X^'^ in ^^([s, T], M), for all p > 2. Moreover, with C 
max{(5C + max a'^{t),SC + max A{t)}, it satisfies the following moment estimate for 

every t G [s, T]; 



s<t<T 



V\„Cp{t-s) 



(h) Let Ut = u{t,Xt) be a Markovian feedback control where u{t,x) is locally Lipshitz with 
respect to x and also satisfies Condition\^ (iv), for some generic constant C > 0. Then the 
SDE has a unique solution in Lp([s,T],M), for all p > 2, too, and a similar moment 
estimate as in part (a) holds. 

(c) If both r{t,x) and u{t,x) satisfy a linear growth condition in x, then the unique solution 
of [24\ ) is in ^^([s, T], M) with the estimate 

Es,. sup |X,^'"r< (l + |xne^i(^-^), 

s<t<T 

where Ci is a constant depending on p, T, s and the linear growth factor. 

Proof, (a) By (iv), for all t e [s,T] and x E R, xfx{t,x) + ^a'^{t) < (A(t) + 5C)x'^ + 
5C + ^a^{t) < C{1 + x^). Since /i(t, x) is also locally Lipshitz, the result follows from the 
standard estimates based on the Lyapunov function approach. See, for example, Mao (1997), 
Ch. 3, Theorem 4.1. The proof of part (b) is similar, replacing /^(t, x) with fi{t,x,u{t,x)), 
and part (c) is a standard result for the SDEs with coefficients of linear growth. 



Lemma 11 Let the assumptions of Condition O hold and u* be an optimal control rule: 
ul = u{t,X^) with V{s,x) = J"*(s,x) and X^ = X^ . Moreover, let X* = x = X^. Then we 
have the following estimates: 

(a) There is a positive constant C such that < V{s,x) < C{1 + x"^), for every x G M, 
uniformly on [0,T]. 

T 

(b) Es^x J \ut\ — ^(1 + 1^1 ) where C is as in part (a) 

s 

(c) Es^x{X^ — XtY < C{1 + \xf), for some positive constant C. 

T 

Proof, (a) For u = 0, smce /(.) > 0, we get V{s,x) < J°(s,x) < Eq,^ /[/(t)(Xt - 



^{t)f)]dt. Since ^(.) and /(.) are bounded on [0,T], the result follows from Lemma [70l with 
p = 2. 
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T 

(b) By Hlj and assumption (u) of CondzUon\g V{s,x) = Es,x j[l{t){Xl-i{t) f+k{t)uf)]dt > 

s 

T 

eEs^x J \ut\ dt. Hence the inequality is a result of part (a). 

s 

(a) First note that d{X; - Xt) = {A{t){X; - Xt) + 6{r{t,X;) - r{t,Xt)) + B{t)ul}dt and 
let A{t) = X* - Xt, for s <t<T. Then by Ito's rule, 

t t t 

A2(t) = j 2A{v)A^{v)dv + J 25{r{t,X;) -r(t,Xt))A{v)dv + j 2B{v)ulA{v)dv. 

s s s 

By applying the monotonicity assumption (v) of ConditionlB to the second integral and the 
inequality 2ab < + b"^ to the third one above, we get 

t 

0<A"(t)< I {{2A{v) + 25K + l)A^{v)dv+ [{B{v)ulfdv, 



where K is as in Condition\^(v). Since B{.) is bounded on [0,T], by Lemma Ul] E^^x J \B{v)ulf dv < 

s 

some positive constant Ci. Moreover, 3M > such that2A{t)+26K+l < M, 

for < t < T . Therefore, by letting g{s,x;t) = g{t) = Es^xA'^{t), we obtain the inequality 

t 

< g{t) < Ci(l + \xf) + M J g{v)dv. By Gronwall's inequality, g{t) < Ci(l + |a;|^)(l + 

s 

t 

M j e^^^^~^^dv) and hence the result follows. ■ 

s 

Now, by an appeal to Lemma [TO] part (a) with p = 2, and part (c) of the Lemma above, 
we get the following Corollary: 

Corollary 12 (a) Es,x |^t1 < + m), for some positive constant Ki. 
(b) Eg^x \Xt\ < K2{1 + |x|), for some positive constant K2. 

Theorem 13 Let the assumptions (i)-(v) hold and consider the perturbed state dynamics 
together with the cost function [2D\] and the value function V{s,x). Then, 

(a) The value function V{s,x) is in C"'^'^([0,T] x M) such that 
(i) it satisfies the HJB equation 

Vt{t,x) + ^a\t)vxx{t,x)+F{t,x,a{t)vx) + 6r{t,x)vx{t,x) = (27) 

v{T,x) = 0, 

where F{t, x, z) = {x - ^{t)f - over [0, T] x R. 

(a) 3C > such that \Vx{s,x)\ < C(l + uniformly for s G [0,T]. 

(b) The process = u{t,Xt) = ^^^Vx{t, Xt) is an admissible feedback control rule such 
that u{s, x) is the unique minimizer of J"(s, x), for all (s, x, u) G [0, T] x M x M. Moreover, 
u{.,x) is a locally Lipshitz function of x, and satisfies a linear growth condition in x (as in 
part (a)(ii) above). 
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Proof, (a) The proof relies on the approximation of the domain [0, T] x M by the com- 
pact subsets, and the standard but lenghty locahzation arguments and passages to the hmit 
for the Cauchy problems of second order parabolic equations, as in Tsai (1978) and Fleming 
and Rishel (1975). We skip these technical details since the steps involved are very similar 
to those of Lemma 3.1 and Lemma 3.2 of Tsai (1978). 

(b) From the more general equations ( ITTj) and ( IT2l) . it is easy to see that u{s, x) = -^^Vxit, x) 
is the unique minimizer of J"(s,x) in fl25l) . It has a linear growth in x by part (a)(ii), and 
is square integrable by Lemma [TTl It is locally Lipshitz in x uniformly in t since is 
bounded on [0, T], and Vx{t,x) is differentiable with respect to x. 

Remark 14 Since no specific growth condition is assumed on the perturbation term r{t,x), 
except the monotonicty and local Lipshitz properties, the standard verification theorems and 
dynamic programming principle of stochastic control theory may not directly apply. Again, 
some localization techniques would help to ensure the uniqueness of the solutions to the PDF 
([F7|)jf| However we follow a BSDE approach to prove the uniqueness. We then provide a 
probabilistic representation of both the value function and the optimal control. 



Theorem 15 Let Xf'^ , X*, u* and V{t,x) be as before, and introduce the pair (F/'^, Z*'"^) 
as Yt = V{t,Xt) and Zt = a(t)K(t,Xt). Then, 

(i) The pair (F/'^, Z*'^) is the unique solution to the BSDE 

dY''"' = -F{t, Xt, Zt)dt + ZtdWt, Y^'"" = 0. (28) 

Moreover, V{t,x) = F/'^ (the value of Yt when Xt = x), and u{t,x) = ^{t)Zl'^ for 
a; > and t G [0,T). An optimal (feedback) control for the problem is given . 

(a) The value function is the unique classical solution of the PDF [2l\ ) with a quadratic 
growth in x. Moreover, the control process u* is the unique optimal control for the optimiza- 
tion problem ^^2^-^2^. 

Proof. The uniqueness follows from the Theorem [6] and the Corollary [HI Then the 
optimality is a direct result of Part (ii) is a result of Theorem US] and the uniqueness. The 
BSDE representation in (i) is obtained similar to that of fll6p - (IT7|) and Remark O by following 
the same steps as in Lemma 0. ■ 

Corollary 16 The triple {X^'^, F/'^, Z^'^) satisfies 

(a) sup E,,, ir^l + EsAjZ^dt + JX^dt] < C(l + x^). 

s<t<T s s 

T 

(b) If the perturbation term r{t,x) has a linear growth, then Eg^x sup \Yt\ + Es^x[J Z^dt + 

s<t<T ' s 

T 

/ X^dt] < C{1 + x^) also holds. 

s 

•^Tsai (1978) take advantage of the differentiability (and tlie existence of an upper bound on the derivative) 
that doesn't apply here. 
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Proof. It is a result of Lemma [TT| Theorem [T3] part (a) (ii), and Theorem [T5| by utihzing 
Lemma [TO] (a) for the proof of part (a), and Lemma [TO] (c) for the proof of part (b), with 
p = 2. m 

The following example is a slightly generalization of an application from Tsai (1978). 
The well-posedness of more general cases are discussed in Cetin (2012a). 

Example 17 Consider the following controlled state dynamics with constant parameters: 

dXt = {-6Xf + But)dt - adWt 
Xo = xo> 0. 

1 

Let the cost functional for this perturbed problem be given by J'^'"(s, x) = E^^^ fli^t ~ 0^ + 

s 

kuf)]dt where k > and the value function is V^{s,x) = inf J'^'"(s, x). By the Theorem 
above, V^{s,x) is the unique solution to the PDE 

vt{t,x) + \7'^v^^{t,x) + {x-if -5x^v^{t,x)-Cvl{t,x)/A = 

v{T,x) = 

where C = B'^/k > over [0,T]. Again, it is not likely to obtain an explicit solution 
of this nonlinear and one needs to follow a numerical procedure to solve the equation and 
hence describe the behavior of the optimal action (control). Note that for the unperturbed 
LQR problem (6 = 0) with the constant parameters C and C,, it can be verified (using the 
results in subsection 2.3) that the solution is given by the quadratic expression V^°(s,x) = 
X{t){x-if+-l{t), where X{t) = ^'^'^Mv^Ci-*)) and-f{t) = ^ lncosh(v^(l-t)). Moreover, the 
optimal control Uf'^ is a linear feedback control: = —^{t)V^{t,x) = ^^^^^^ tanh(|^(l — 
t)){x — C,) . When the state equation deviates from a linear dynamics significantly, then it 
may be important to know how the corresponding optimal action and the value function differ 
from the unperturbed LQR setup. The discussion of this problem and its numerical solution 
using a probabilistic appraoch are considered in Cetin (2012b). One can also refer to Tsai 
(1978) and Nishikawa et al. (1976) for a PDE approach for the details. Some results are 
given below without proof: 

• The value function V^{s,x) can be approximated as follows: 

V\s, x) = V\s, x) + 26[Ki{s)x* + K2{s)x^] + 0{6^) 

where V^{s,x) is the value function for the unperturbed problem and Ki{.),K2{.) G 
C'[0,1]. 

• The optimal control is 

u*'^{s, x) = u*'%s, x) - 26[AKi{s)x^ + 2K2{s)x^] + 0{6^) 
where u*'^{s,x) is as above. 

• u*'^{s,x) — > u*''^{s,x), as 5 — !■ uniformly on [0,T] x Q, for any compact set Q o/M. 
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